{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benign Evaluation\n",
    "\n",
    "Benign Evaluation is testing the performance of a model on benign data. This is the traditional way of evaluating a model. The model is trained on a training set and evaluated on a test set. The test set is a set of data that the model has not seen before. The model is evaluated on how well it generalizes to unseen data. `advsecurenet` supports benign evaluation using the `test` command. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a Configuration File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Create a Configuration File\n",
    "\n",
    "As a first step, you need to have a default configuration file. You can create a configuration file by running the following command:\n",
    "    \n",
    "```sh\n",
    "advsecurenet utils configs get -c test_config.yml -s -o ./default_test_config.yml\n",
    "```\n",
    "\n",
    "Here, we are also providing the `-o` flag to specify the output directory where the configuration file will be saved. You can also specify the output directory by passing the `-o` flag followed by the path to the directory where you want to save the configuration file. If you don't provide the `-o` flag, the configuration file will be saved in the current working directory with the name `test_config.yml`. If the file already exists, it will be overwritten. If the output path is a directory, the configuration file will be saved in that directory with the name `test_config.yml`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving default config to ./default_test_config.yml\n",
      "Generated default configuration file test_config.yml!\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet utils configs get -c test_config.yml -s -o ./default_test_config.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1m**************************************************\u001b[0m\n",
      "\u001b[31m\u001b[1mDefault configuration file for the test_config.yml:\n",
      "\u001b[0m\n",
      "\u001b[32m{\n",
      "    \"model\": {\n",
      "        \"model_name\": \"resnet18\",\n",
      "        \"num_input_channels\": 3,\n",
      "        \"num_classes\": 10,\n",
      "        \"pretrained\": true,\n",
      "        \"weights\": \"IMAGENET1K_V1\",\n",
      "        \"is_external\": false,\n",
      "        \"random_seed\": null,\n",
      "        \"path_configs\": {\n",
      "            \"model_arch_path\": null,\n",
      "            \"model_weights_path\": null\n",
      "        },\n",
      "        \"norm_config\": {\n",
      "            \"add_norm_layer\": false,\n",
      "            \"norm_mean\": [\n",
      "                0.5,\n",
      "                0.5,\n",
      "                0.5\n",
      "            ],\n",
      "            \"norm_std\": [\n",
      "                0.5,\n",
      "                0.5,\n",
      "                0.5\n",
      "            ]\n",
      "        }\n",
      "    },\n",
      "    \"dataset\": {\n",
      "        \"dataset_name\": \"cifar10\",\n",
      "        \"num_classes\": 10,\n",
      "        \"train_dataset_path\": null,\n",
      "        \"test_dataset_path\": null,\n",
      "        \"preprocessing\": {\n",
      "            \"steps\": [\n",
      "                {\n",
      "                    \"name\": \"ToImage\",\n",
      "                    \"params\": null\n",
      "                },\n",
      "                {\n",
      "                    \"name\": \"ToDtype\",\n",
      "                    \"params\": {\n",
      "                        \"dtype\": \"torch.float32\",\n",
      "                        \"scale\": true\n",
      "                    }\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"download\": true\n",
      "    },\n",
      "    \"dataloader\": {\n",
      "        \"test\": {\n",
      "            \"num_workers\": 4,\n",
      "            \"shuffle\": true,\n",
      "            \"drop_last\": false,\n",
      "            \"pin_memory\": true,\n",
      "            \"batch_size\": 32\n",
      "        }\n",
      "    },\n",
      "    \"device\": {\n",
      "        \"use_ddp\": false,\n",
      "        \"processor\": \"cpu\",\n",
      "        \"gpu_ids\": null\n",
      "    },\n",
      "    \"testing\": {\n",
      "        \"criterion\": \"cross_entropy\",\n",
      "        \"topk\": 1\n",
      "    }\n",
      "}\u001b[0m\n",
      "\u001b[34m\u001b[1m**************************************************\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Let's check the content of the default config file\n",
    "!advsecurenet utils configs get -c test_config.yml -p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Modify the Configuration File\n",
    "\n",
    "The default configuration file is capable of running the benign evaluation on the `CIFAR10` dataset using the `ResNet18` model. You can modify the configuration file to run the evaluation on a different dataset, model, or change other hyperparameters. You can also add new hyperparameters to the configuration file.\n",
    "\n",
    "\n",
    "For the sake of this example, we will modify the configuration file to run the top-1 and top-5 accuracy on the `CIFAR10` dataset using the `ResNet18` model pre-trained on the `ImageNet` dataset. \n",
    "\n",
    "The updated configuration file can be found [here](./test_config.yml).\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Test the Model\n",
    "\n",
    "Now, you can test the model by running the following command:\n",
    "\n",
    "```sh\n",
    "advsecurenet evaluate test -c test_config.yml\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n",
      "Testing: 100%|█████████████████████████████| 313/313 [00:46<00:00,  6.72batch/s]\n",
      "\n",
      "Test set: Average loss: 0.0741, Top-1 Accuracy: 1077/10000 (10.77%)\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet evaluate test -c test_config.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n",
      "Testing: 100%|█████████████████████████████| 313/313 [00:44<00:00,  7.08batch/s]\n",
      "\n",
      "Test set: Average loss: 0.0739, Top-5 Accuracy: 4783/10000 (47.83%)\n"
     ]
    }
   ],
   "source": [
    "# We can also provide arguments to the command line which will override the default config file\n",
    "# Now we would like to evaluate the top-5 accuracy\n",
    "!advsecurenet evaluate test -c test_config.yml --topk 5"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adv2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
